Positional shift amount calculation apparatus and imaging apparatus

ABSTRACT

A positional shift amount calculation apparatus that calculates a positional shift amount, which is a relative positional shift amount between a first image, based on a luminous flux that has passed through a first imaging optical system, and a second image. A calculation unit calculates a positional shift amount based on data within a predetermined area out of first image data representing first and second image data. A setting unit sets a relative size of the area to the first and second image data. The calculation unit calculates a first positional shift amount using the first and second image data in the area having a first size that is preset. The setting unit sets a second size of the area based on the size of the first positional shift amount and an optical characteristic of the first imaging optical system. The calculation unit then calculates a second positional shift amount.

CLAIM TO PRIORITY

This application is a national stage application of International PatentApplication No. PCT/JP2015/004474, filed Sep. 3, 2015, which claims thebenefit of Japanese Patent Application No. 2014-188472, filed on Sep.17, 2014, and Japanese Patent Application No. 2015-155151, filed on Aug.5, 2015, which are hereby incorporated by reference herein in theirentireties.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to a technique to calculate the positionalshift amount between images.

Description of the Related Art

A known depth measuring apparatus measures depth by calculating apositional shift amount (also called “parallax”), which is a relativepositional shift amount between two images having different points ofview (hereafter called “image A” and “image B”). To calculate thepositional shift amount, an area-based corresponding points searchtechnique called “template matching” is often used. In templatematching, either image A or image B is set as a base image, and theother image, which is not the base image, is set as a reference image. Abase area around a target point (also called “base window”) is set onthe base image, and a reference area around a reference pointcorresponding to the target point (also called “reference window”) isset on the reference image. The base area and the reference area arecollectively called “matching windows”. A reference point at which thesimilarity of an image in the base area and an image in the referencearea is highest (correlation thereof is highest) is searched for whilesequentially moving the reference point, and the positional shift amountis calculated using the relative positional shift amount between thetarget point and the reference point. Generally, a calculation erroroccurs to the positional shift amount due to a local mathematicaloperation if the size of the base area is small. Hence, a relativelylarge area size is used.

The depth (distance) to an object can be calculated by converting thepositional shift amount into a defocus amount or into an object depthusing a conversion coefficient. This allows measuring the depth athigh-speed and at high accuracy, since it is unnecessary to move thelens to measure the depth.

The depth measurement accuracy improves by accurately determining thepositional shift amount. Factors that cause an error to the positionalshift amount are changes of the positional shift amount in each pixel ofthe base area, and noise generated in the process of acquiring imagedata. To minimize the influence of the changes of the positional shiftamount in the base area, the base area must be small. If the base areais small, however, a calculation error to the positional shift amountmay be generated by the influence of noise or because of the existenceof similar image patterns.

In Japanese Patent Application Laid-Open No. 2011-013706, the positionalshift amount is calculated for each scanning line (e.g., a horizontalline), and the positional shift amount at the adjacent scanning line iscalculated based on the calculated positional shift amount data. In thiscase, a method of setting a base area independently for each pixel, sothat a boundary where the calculated positional shift amount changes isnot included, has been proposed.

In Japanese Patent Application Laid-Open No. H10-283474, a method ofdecreasing the size of the base area in steps and gradually limiting thesearch range to search for a corresponding point is proposed.

SUMMARY OF THE INVENTION

A problem of the positional shift amount calculation method disclosed inJapanese Patent Application Laid-Open No. 2011-013706, however, is thatthe memory amount and computation amount required for calculating thepositional shift amount are large. This is because the positional shiftamount of a spatially adjacent area is calculated and evaluated inadvance to determine the size of the base area. Furthermore, in a casewhen the positional shift amount changes continuously, the base areabecomes small since the base area is set in a range where the positionalshift amount is approximately the same, and a calculation error mayoccur to the positional shift amount. In other words, the depth may bemiscalculated depending on the way of changing the object depth.

A problem of the positional shift amount calculating method disclosed inJapanese Patent Application Laid-Open No. H10-283474 is that thecomputation amount is large since a plurality of base areas is set ateach pixel position, and the correlation degree is evaluated.

With the foregoing in view, it is an object of the present invention toprovide a technique that can calculate the positional shift amount athigh accuracy by an easy operation.

Solution to Problem

A first aspect of the present invention is to provide a positional shiftamount calculation apparatus that calculates a positional shift amount,which is a relative positional shift amount between a first image basedon a luminous flux that has passed through a first imaging opticalsystem, and a second image, the apparatus having a calculation unitadapted to calculate the positional shift amount based on data within apredetermined area out of first image data representing the first imageand second image data representing the second image, and a setting unitadapted to set a relative size of the area to the first and second imagedata, and in this positional shift amount calculation apparatus, thecalculation unit being adapted to calculate a first positional shiftamount using the first image data and the second image data in the areahaving a first size which is preset, the setting unit being adapted toset a second size of the area based on the size of the first positionalshift amount and an optical characteristic of the first imaging opticalsystem, and the calculation unit being adapted to calculate a secondpositional shift amount using the first image data and the second imagedata in the area having the second size.

A second aspect of the present invention is to provide a positionalshift amount calculation method for a positional shift amountcalculation apparatus to calculate a positional shift amount, which is arelative positional shift amount between a first image based on aluminous flux that has passed through a first imaging optical system,and a second image, the method having a first calculation step ofcalculating a first positional shift amount based on data within an areahaving a predetermined first size, out of first image data representingthe first image and second image data representing the second image, asetting step of setting a second size, which is a relative size of thearea to the first and second image data, based on the size of the firstpositional shift amount and an optical characteristic of the firstimaging optical system, and a second calculation step of calculating asecond positional shift amount using the first image data and the secondimage data in the area having the second size.

Advantageous Effects of Invention

According to the present invention, the positional shift amount can becalculated at high accuracy by an easy operation.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A and FIG. 1B are diagrams depicting a configuration of a digitalcamera that includes a depth calculation apparatus.

FIG. 2 is a diagram depicting a luminous flux that the digital camerareceives.

FIG. 3 is a flow chart depicting a depth calculation procedure accordingto the first embodiment.

FIG. 4A to FIG. 4D are diagrams depicting a positional shift amountcalculation method and a factor that generates a positional shift amounterror.

FIG. 5A and FIG. 5B are diagrams depicting a base line length.

FIG. 6A and FIG. 6B are diagrams depicting a depth calculation unitaccording to the first embodiment.

FIG. 6C to FIG. 6F are diagrams depicting a size of a second base areaaccording to the first positional shift amount.

FIG. 7A to FIG. 7D are diagrams depicting the reason why the positionalshift amount can be accurately calculated in the first embodiment.

FIG. 8A and FIG. 8B are diagrams depicting the depth calculation unitaccording to a modification.

FIG. 8C to FIG. 8E are diagrams depicting the depth calculation unitaccording to a modification.

FIG. 9 is a flow chart depicting a general operation of the digitalcamera.

FIG. 10 is a diagram depicting a configuration of the digital cameraaccording to the modification.

FIG. 11 is a flow chart depicting an example of the positional shiftamount calculation procedure according to the first embodiment.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Embodiments of the present invention will now be described withreference to the drawings. In the following description, a digitalcamera is described as an example of an imaging apparatus that includesa depth calculation apparatus (positional shift calculation apparatus),but application of the present invention is not limited to this. Forexample, the positional shift amount calculation apparatus of thepresent invention can be applied to a digital depth measuringinstrument.

In the description with reference to the drawings, as a rule, a samesegment is denoted by a same reference number, even if the figure numberis different, and a redundant description is minimized.

First Embodiment

<Configuration of a Digital Camera>

FIG. 1A is a diagram depicting a configuration of a digital camera 100that includes a depth measurement apparatus. In the digital camera 100,an imaging optical system 120, an imaging element 101, a depthcalculation unit 102, an image storage unit 104, an image generationunit (not illustrated), a lens driving control unit (not illustrated),and a control unit (not illustrated) are disposed inside a camera casing130. The imaging optical system 120, the imaging element 101, the depthcalculation unit 102, and the image storage unit 104 constitute a depthcalculation apparatus 110. The depth calculation unit 102 can beconstructed using a logic circuit. As another mode, the depthcalculation unit 102 may be constituted by a central processing unit(CPU) and a memory storing arithmetic processing programs. The depthcalculation unit 102 corresponds to the positional shift amountcalculation apparatus according to the present invention.

The imaging optical system 120 is a photographing lens of the digitalcamera 100, and has a function to form an image of the object on theimaging element 101, which is an imaging surface. The imaging opticalsystem 120 is constituted by a plurality of lens groups (notillustrated) and a diaphragm (not illustrated), and has an exit pupil103 at a position apart from the imaging element 101 by a predetermineddistance. Reference number 140 in FIG. 1A denotes an optical axis of theimaging optical system 120, and, in this description, the optical axisis assumed to be parallel with the z axis. The x axis and the y axis areassumed to be orthogonal to each other, and are orthogonal to theoptical axis.

An operation example of this digital camera 100 will now be describedwith reference to FIG. 9. The following is merely an example, andoperation of the digital camera 100 is not limited to this example. FIG.9 is a flow chart depicting an operation flow after the main power ofthe digital camera 100 is turned ON and the shutter button (notillustrated) is half depressed. First, in step S901, the control unitreads the information on the imaging optical system 120 (e.g., focallength, diaphragm value), and stores the information in the memory unit(not illustrated). Then, the control unit executes the processing insteps S902, S903, and S904 to adjust the focal point. In other words, instep S902, the depth calculation unit 102 calculates a defocus amountusing the depth calculation procedure shown in FIG. 3, based on theimage data outputted from the imaging element 101. The depth calculationprocedure will be described in detail later. In step S903, the controlunit determines whether the imaging optical system 120 is in the focusedstate or not based on the calculated defocus amount. If not focused, thecontrol unit drives the imaging optical system 120 to the focusedposition based on the defocus amount using the lens driving controlunit, and then, processing returns to step S902. If it is determinedthat the imaging optical system 120 is in the focused state in stepS903, the control unit determines whether the shutter was released(fully depressed) by the operation of the shutter button (notillustrated) in step S905. If not released, processing returns to stepS902, and the above mentioned processing is repeated. If it isdetermined that the shutter is released in step S905, the control unitreads image data from the imaging element 101, and stores the image datain the image storage unit 104. The image generation unit performsdevelopment processing on the image data stored in the image storageunit 104, whereby, a final image can be generated. Further, an objectdepth image (object depth distribution) corresponding to the final imagecan be generated by applying the depth calculation procedure, which willbe described later with reference to FIG. 3, to the image data stored inthe image storage unit 104.

<Configuration of Imaging Element>

The imaging element 101 is constituted by a CMOS (ComplementaryMetal-Oxide Semiconductor) or a CCD (Charge-Coupled Device). The objectimage is formed on the imaging element 101 via the imaging opticalsystem 120, and the imaging element 101 performs photoelectricconversion on the received luminous flux, and generates image data basedon the object image. The imaging element 101 according to thisembodiment will now be described in detail with reference to FIG. 1B.

FIG. 1B is an xy cross-sectional view of the imaging element 101. Theimaging element 101 has a configuration in which a plurality of pixelgroups (2 rows×2 columns) is arranged. Each pixel group 150 isconstituted by green pixels 150G1 and 150G2, which are disposed indiagonal positions, and a red pixel 150R and a blue pixel 150B, whichare the other two pixels.

<Principle of Depth Measurement>

In each pixel constituting the pixel group 150 of this embodiment, twophotoelectric conversion units (first photoelectric conversion unit 161,and second photoelectric conversion unit 162), of which shapes aresymmetric in the xy cross section, are disposed in the light receivinglayer (203 in FIG. 2) in the pixel. The luminous flux received by thefirst photoelectric conversion unit 161 and the second photoelectricconversion unit 162 in the imaging element 101 will be described withreference to FIG. 2.

FIG. 2 is a schematic diagram depicting only the exit pupil 103 of theimaging optical system 120 and the green pixel 150G1 as an examplerepresenting the pixels disposed in the imaging element 101. The pixel150G1 shown in FIG. 2 is constituted by a color filter 201, a micro lens202 and a light receiving layer 203, and the first photoelectricconversion unit 161 and the second photoelectric conversion unit 162 areincluded in the light receiving layer 203. The micro lens 202 isdisposed such that the exit pupil 103 and the light receiving layer 203are in a conjugate relationship. As a result, the luminous flux 210 thathas passed through a first pupil area (261) of the exit pupil enters thefirst photoelectric conversion unit, and the luminous flux 220 that haspassed through a second pupil area (262) thereof enters thephotoelectric conversion unit 162, as shown in FIG. 2.

The plurality of first photoelectric conversion units 161 disposed ineach pixel performs photoelectric conversion on the received luminousflux and generates the first image data. In the same manner, theplurality of second photoelectric conversion units 162 disposed in eachpixel performs photoelectric conversion on the received luminous fluxand generates the second image data. From the first image data, theintensity distribution of the first image (image A), which the luminousflux that has mainly passed through the first pupil area forms on theimaging element 101, can be acquired. From the second image data, theintensity distribution of the second image (image B), which the luminousflux that has mainly passed through the second pupil area forms on theimaging element 101, can be acquired. Therefore, the relative positionalshift amount of the first image and the second image is the positionalshift amount of the image A and image B. By calculating this positionalshift amount according to a later mentioned method and converting thecalculated positional shift amount into the defocus amount using aconversion coefficient, the depth (distance) to the object can becalculated.

<Description on Depth Calculation Procedure>

The depth calculation procedure of this embodiment will now be describedwith reference to FIG. 3.

In step S1, the imaging element 101 acquires the first image data andthe second image data, and transfers the acquired data to the depthcalculation unit 102.

In step S2, the light quantity balance correction processing isperformed to correct the balance of the light quantity between the firstimage data and the second image data. To correct the light quantitybalance, a known method can be used. For example, a coefficient tocorrect the light quantity balance between the first image data and thesecond image data is calculated based on an image acquired byphotographing a uniform surface light source in advance using thedigital camera 100.

In step S3, the depth calculation unit 102 calculates the positionalshift amount based on the first image data and the second image data.The calculation method for the positional shift amount will be describedlater with reference to FIG. 4A to FIG. 4D and FIG. 6A to FIG. 6F.

In step S4, the depth calculation unit 102 converts the positional shiftamount into an image-side defocus amount using a predeterminedconversion coefficient. The image-side defocus amount is a distance froman estimated focal position (imaging element surface) to the focalposition of the imaging optical system 120.

The calculation method for the conversion coefficient that is used forconverting the positional shift amount into the image side defocusamount will now be described with reference to FIG. 5A and FIG. 5B. FIG.5A shows a light receiving sensitivity incident angle characteristic ofeach pixel. The abscissa indicates the incident angle of the light thatenters the pixel (angle formed by the ray projected to the xz plane andthe z axis), and the ordinate indicates the light receiving sensitivity.The solid line 501 indicates the light receiving sensitivity of thefirst photoelectric conversion unit, and the broken line 502 indicatesthe light receiving sensitivity of the second photoelectric conversionunit.

FIG. 5B shows the light receiving sensitivity distribution on the exitpupil 103 when this receiving sensitivity is projected onto the exitpupil 103. The darker the color, the higher the light receivingsensitivity. In FIG. 5B, reference numeral 511 indicates a center ofgravity position of the light receiving sensitivity distribution of thefirst photoelectric conversion unit, and reference numeral 512 indicatesa center of gravity position of the light receiving sensitivitydistribution of the second photoelectric conversion unit. The distance513 between the center of gravity position 511 and the center of gravityposition 512 is called “base line length”, and is used as the conversioncoefficient to convert the positional shift amount into the image sidedefocus amount. When r is the positional shift amount, w is the baseline length and L is the pupil distance from the imaging element 101 tothe exit pupil 103, the positional shift amount can be converted intothe image side defocus amount ΔL using the following Expression 1.

$\begin{matrix}{{\Delta\; L} = \frac{r \cdot L}{w - r}} & (1)\end{matrix}$

In this embodiment, the positional shift amount is converted into theimage-side defocus amount using Expression 1, but the positional shiftamount may be converted into the image side defocus amount by adifferent method. For example, based on the assumption that the baseline length w is sufficiently larger than the positional shift amount rin Expression 1, a gain value Gain may be calculated using Expression 2,and the positional shift amount may be converted into the image sidedefocus amount based on Expression 3.Gain=L/w  (2)ΔL=Gain·r  (3)

By using Expression 3, the positional shift amount can be easilyconverted into the image side defocus amount, and the computation amountto calculate the object depth can be reduced. A lookup table forconversion may be used to convert the positional shift amount into theimage side defocus amount. In this case, as well, the computation amountto calculate the object depth can be reduced.

In FIG. 2, it is assumed that x is positive in the first pupil area, andx is negative in the second pupil area. However, the actual light thatreaches the light receiving layer 203 has a certain spread due to thelight diffraction phenomenon, and, therefore, the first pupil area andthe second pupil area overlap, as shown in the light receivingsensitivity distribution in FIG. 5B. For convenience, however, the firstpupil area 261 and the second pupil area 262 are assumed to be clearlyseparated in the description of this embodiment.

In step S5, the image side defocus amount calculated in step S4 isconverted into the object depth based on the image forming relationshipof the imaging optical system (object depth calculation processing).Conversion into the object depth may be performed by a different method.For example, the image side defocus amount is converted into theobject-side defocus amount, and the sum of the object side defocusamount and the object-side focal position, which is calculated based onthe focal length of the imaging optical system 120, is calculated,whereby the depth to the object is calculated. The object-side defocusamount can be calculated using the image-side defocus amount and thelongitudinal magnification of the imaging optical system 120.

In the depth calculation procedure of this embodiment, the positionalshift amount is converted into the image-side defocus amount in step S4,and then, the image-side defocus amount is converted into the objectdepth in step S5. However, the processing executed after calculating thepositional shift amount may be other than the above mentionedprocessing. As mentioned above, the image-side defocus amount and theobject-side defocus amount, or the image-side defocus amount and theobject depth can be converted into each other using the image formingrelationship of the imaging optical system 120. Therefore, thepositional shift amount may be directly converted into the object-sidedefocus amount or the object depth, without being converted into theimage-side defocus amount. In either case, the defocus amount(image-side and/or object-side) and the object depth can be accuratelycalculated by accurately calculating the positional shift amount.

In this embodiment, the image-side defocus amount is converted into theobject depth in step S5, but step S5 need not always be executed, andthe depth calculation procedure may be completed in step S4. In otherwords, the image-side defocus amount may be the final output. The bluramount of the object in the final image depends on the image-sidedefocus amount, and, as the image-side defocus amount of the objectbecomes greater, a more blurred image is photographed. To performrefocusing processing for adjusting the focal position in the imageprocessing in a subsequent step, it is sufficient if the image-sidedefocus amount is known, and conversion into the object depth isunnecessary. As mentioned above, the image-side defocus amount can beconverted into/from the object side defocus amount or the positionalshift amount. Hence, the final output may be the object-side defocusamount or the positional shift amount.

<Factor for Generating Positional Shift Amount Error>

A calculation method for the positional shift amount will be describedfirst with reference to FIG. 4A to FIG. 4D. FIG. 4A is a diagramdepicting the calculation method for the positional shift amount, wherethe first image data 401, second image data 402, and photographingobject 400 are shown. A target point 410 is set for the first image data401, and a base area 420 is set centering around the target point 410.For the second image data 402, on the other hand, a reference point 411is set at a position corresponding to the target point 410, and thereference area 421 is set centering around the reference point 411. Thesizes of the base area 420 and the reference are 421 are the same. Whilethe reference point 411 is sequentially moved within a predeterminedpositional shift amount searching range, a correlation value between thefirst image data in the base area 420 and the second image data in thereference area 421 is calculated, and the reference point 411, at whichthe highest correlation value is acquired, is regarded as being acorresponding point of the target point 410. The positional shift amountsearching range is determined based on the maximum depth and the minimumdepth to calculate. For example, the maximum depth is set to infinity,and the minimum depth is set to the minimum photographing depth of theimaging optical system 120, and the range of the maximum positionalshift amount and the minimum positional shift amount, which aredetermined by the maximum depth and the minimum depth respectively, isset as the positional shift amount searching range. The positional shiftamount is a relative positional shift amount between the target point410 and the corresponding point. By searching for the correspondingpoint while sequentially moving the target point 410, the positionalshift amount at each data position (each pixel position) in the firstimage data can be calculated. To calculate the correlation value, aknown method can be used, such as the SSD method, where a square-sum ofthe difference between each pixel data (each pixel value) in the basearea 420 and each pixel data in the reference area 421 is used as anevaluation value.

Now, a factor that generates the positional shift amount error will bedescribed. In FIG. 4B, the abscissa indicates the positional shiftamount, and the ordinate indicates the correlation degree evaluationvalue based on SSD. A curve that indicates the correlation degreeevaluation value for each positional shift amount is hereafter called a“correlation value curve”. The solid line in FIG. 4B indicates thecorrelation value curve when the contrast of the object image is high,and the broken line indicates the correlation value curve when thecontrast of the object image is low. The correlation value curve has aminimum value 430. The positional shift amount, where the correlationdegree evaluation value is the minimum value, is determined as thepositional shift amount of which correlation is highest, that is, isdetermined as the positional shift amount. As the correlation valuecurve changes more sharply, the influence of noise decreases. Therefore,a calculation error of the positional shift amount decreases. Hence, ifthe contrast of the object image is high, the positional shift amountcan be accurately calculated. If the contrast of the object image islow, on the other hand, the change of the correlation value curvebecomes gentle. Hence, a calculation error of the positional shiftamount increases.

Contrast does not deteriorate very much in an area near the focalposition of the imaging optical system 120. Hence, a high contrastobject image can be acquired near the focal position. As the position ofthe object moves away from the focal position of the imaging opticalsystem 120 (as the object is defocused), the contrast drops, and thecontrast of the acquired image also decreases. If the defocus amount isplotted on the abscissa and the positional shift amount error is plottedon the ordinate, as shown in FIG. 4C, the positional shift amount errorincreases as the defocus amount increases.

If the positional shift amount changes within the base area, a bimodalcorrelation value curve, having two minimum values, is acquired, asshown in FIG. 4D. In this case, the positional shift amount iscalculated based on the smaller of the two minimum values, which meansthat the positional shift amount may be miscalculated.

<Detailed Description on Positional Shift Amount Calculation Method>

The depth calculation unit 102 of this embodiment and the positionalshift amount calculation procedure S3 will now be described in detailwith reference to FIG. 6A to FIG. 6C. FIG. 6A is a diagram depicting adetailed configuration of the depth calculation unit 102, and FIG. 6B isa flow chart depicting the positional shift amount calculationprocedure.

The depth calculation unit 102 is constituted by a positional shiftamount calculation unit 602, a base area setting unit 603, and a depthconversion unit 604. The positional shift amount calculation unit 602calculates the positional shift amount of the first image data and thesecond image data stored in the image storage unit 104 using a base areahaving a predetermined size, or a base area having a size set by thebase area setting unit 603. The base area setting unit 603 receives thepositional shift amount (first positional shift amount) from thepositional shift amount calculation unit 602, and outputs the size ofthe base area corresponding to this positional shift amount to thepositional shift amount calculation unit 602. The first image data andthe second image data, on which light quantity balance correction hasbeen performed, as described with reference to step S2 in FIG. 3, arestored in the image storage unit 104.

In step S3-1 in FIG. 6B, the positional shift amount calculation unit602 calculates the first positional shift amount based on the firstimage data and the second image data acquired from the image storageunit 104. In concrete terms, the first positional shift amount iscalculated by the corresponding point search method described withreference to FIG. 4A to FIG. 4D, using the base area (first base area)having a size that is set in advance (first size).

In step S3-2 in FIG. 6B, the base area setting unit 603 sets a size ofthe second base area (second size) based on the first positional shiftamount. According to this embodiment, a second base area, of which areasize is greater than the first base area, is set when the absolute valueof the first positional shift amount acquired by the positional shiftamount calculation unit 602 exceeds a predetermined threshold. FIG. 6Cshows the relationship between the absolute value of the firstpositional shift amount and the size of the second base area.

In FIG. 6C, the abscissa indicates the absolute value of the firstpositional shift amount, and the ordinate indicates the size of thesecond base area. The broken line 620 parallel with the abscissaindicates the area size of the first base area (first size). If theabsolute value of the first positional shift amount is greater than thethreshold 610, the size of the second base area (second size) becomeslarger than the area size of the first base area (first size).

In step S3-3 in FIG. 6B, the positional shift amount calculation unit602 searches for a corresponding point again using the second base area,and calculates the second positional shift amount. According to thisembodiment, if the absolute value of the first positional shift amountis a predetermined threshold or less, the positional shift amountcalculation unit 602 does not recalculate the positional shift amount,and regards the first positional shift amount as the second positionalshift amount.

By the above processing, the positional shift amount calculationprocedure S3 completes. Then, the depth conversion unit 604 converts thesecond positional shift amount into the object depth by the methoddescribed in step S4 and S5 in FIG. 3, and outputs the object depthinformation.

<Reason why Changes of Positional Shift Amount and Influence of Noisecan be Reduced>

The reason why changes of the positional shift amount in the base areaand influence of noise generated upon acquiring image signals can bereduced by the depth calculation method executed by the depthcalculation unit 102 of this embodiment will be described with referenceto FIG. 7A to FIG. 7D.

FIG. 7A is a diagram depicting acquisition of the images of an object701 and an object 702 in the digital camera 100. Here, the object 701 isdisposed in a focal position of the imaging optical system 120, and theblur size 711 on the imaging element 101 is small. The object 702, onthe other hand, is disposed in a position distant from the focalposition of the imaging optical system 120, and the blur size 712 on theimaging element 101 is larger. If the object 701 and the object 702 havea brightness distribution shown in FIG. 7B, then, the image of theobject 701 becomes like that shown in FIG. 7C (with little blur), andthe image of the object 702 becomes like that shown in FIG. 7D (withconsiderable blur).

If the defocus amount is large, as in the case of the object 702 (FIG.7D), the acquired image is considerably blurred. Hence, the positionalshift amount gently changes. Further, the acquired object image has alow contrast. Therefore, in order to reduce an error of the positionalshift amount, it is preferable to set a large base area. If the defocusamount is small, as in the case of the object 701 (FIG. 7C), theacquired image is not blurred very much. Hence, the positional shiftamount may sharply change. Further, the acquired object image has high acontrast. Therefore, in order to reduce an error in the positional shiftamount, it is preferable to set a small base area.

If the absolute value of the first positional shift amount is large(that is, if the defocus amount is large), the depth calculation unit102 of this embodiment sets a large base area (second base area), andcalculates the positional shift amount again. In other words, if thecontrast of the image acquired via the imaging optical system 120 islow, and the changes of the positional shift amount are gentle, a largebase area is set.

If the absolute value of the first positional shift amount is greaterthan a predetermined threshold, the depth calculation unit 102 of thisembodiment sets a larger second base area. This makes it unnecessary tocalculate the positional shift amount of spatially adjacent pixels, andboth reducing the influence of the changes of positional shift amountand the influence of noise generated upon acquiring image signals can beimplemented by a simple operation. Furthermore, the base area is setaccording to the optical characteristic of the imaging optical system120. Hence, dependency of the object depth on the changes of thepositional shift amount can be reduced, and the object depth can beaccurately measured.

The depth calculation unit 102 of this embodiment sets a larger size forthe second base area when the absolute value of the first positionalshift amount is greater than a predetermined threshold 610, as shown inFIG. 6C, but the size of the second base area may be determined by adifferent method. For example, as shown in FIG. 6D, the size of thesecond base area may be determined using a linear function of theabsolute value of the first positional shift amount. Or, as a morestandard approach, the size of the second base area may be determined,not based on the absolute value of the first positional shift amount,but based on the first positional shift amount itself. Furthermore,considering the case when the object is closer to the digital camera 100than the focal position, and the case when the object is more distantfrom the digital camera 100 than the focal position, the inclination inthe graph may be changed depending on whether the first positional shiftamount is 0 or more (solid line 630) or the first positional shiftamount is smaller than 0 (broken line 640), as shown in FIG. 6E. As therelationship between the defocus amount and the error of the positionalshift amount is shown in FIG. 4C, the error of the positional shiftamount quadratically increases as the defocus amount increases.Therefore, only when the absolute value of the first positional shiftamount exceeds the threshold 650, the second area size may be set as anincreasing function so that the error of the defocus amount is confinedwithin a predetermined target value, as shown in FIG. 6E. In any case,the changes of the positional shift amount in the base area and theinfluence of the noise generated upon acquiring image signals can bereduced by setting the size of the second base area considering theoptical characteristic of the imaging optical system 120.

The depth calculation unit 102 of this embodiment need not calculate thefirst positional shift amount by setting the target point 410 for all ofthe pixel positions in the first image data. The depth calculation unit102 may calculate the first positional shift amount by sequentiallymoving the target point 410 by a predetermined space. For example, thefirst positional shift amount is calculated by keeping ten pixels ofspace in the horizontal direction and vertical direction, andtwo-dimensional distribution of the first positional shift amount isexpanded by a known expansion method (e.g., bilinear interpolation,nearest neighbor interpolation), and is referred to in order to set thesecond base area. By decreasing a number of target points that are setfor calculating the first positional shift amount, a computation amountrequired for calculating the first positional shift amount can bereduced.

In the present embodiment, the changes of the positional shift amount inthe base area and the influence of noise generated upon acquiring theimage signals are reduced by setting the size of the second base areaconsidering the optical characteristic of the imaging optical system120. Therefore, it is only required that the ratio of the surface area,included in the base area in the first image data, can be changed to setthe size of the base area. If the size of the base area is enlarged toincrease the number of pixels included in the base area, the influenceof the noise generated upon acquiring the image signals can be reduced.Further, the influence of the noise generated upon acquiring the imagesignals can also be suppressed by reducing (thinning out) the image datawhile keeping the number of pixels included in the base area constant.The influence of the noise generated upon acquiring the image signalscan be reduced either by increasing a number of pixels included in thebase area, or by reducing the image data while keeping the number ofpixels included in the base area constant.

In the case of reducing the image data while keeping the number ofpixels included in the base area constant, the positional shift amountcalculation unit 602 shown in FIG. 6A uses the positional shift amountcalculation procedure shown in FIG. 11. In step S3-6, the sizes of thefirst image and the second image are changed using a reduction ratio inaccordance with the size of the second base area. For example, if thesize of the second base area is double the size of the first base area(double in the horizontal direction and vertical direction respectively,four times in terms of area), 0.5 is set as the reduction ratio (0.5times in the horizontal and vertical directions respectively, ¼ times interms of number of pixels). To change the size of the image, a knownmethod, such as the bilinear method, can be used. In step S3-3, by usingthe first image (first reduced image) and the second image (secondreduced image) after the size change, the second positional shift amountis calculated based on the second base area, of which number of pixelsincluded in the base area is the same as the first base area. In FIG.11, the reduction ratio is set in accordance with the size of the secondbase area, but the base area setting unit 603 may output the reductionratio of the image to the positional shift amount calculation unit 602in accordance with the positional shift amount (first positional shiftamount) received from the positional shift amount calculation unit 602.

To calculate the second positional shift amount, both the size of thebase area and the size of the image data may be changed. In other words,the processing executed by the base area setting unit 603 may not belimited to a specific manner only as long as the relative sizes of thebase areas to the first image data and the second image data arechanged. The influence of noise can be reduced if the relative sizes ofthe base areas to the first and second image data upon calculating thesecond positional shift amount are larger than the relative sizes uponcalculating the first positional shift amount.

<Modification 1 of Depth Calculation Unit>

The configuration shown in FIG. 8A may be used as a modification of thedepth calculation unit 102 of this embodiment. The depth calculationunit 102 in FIG. 8A includes a Point Spread Function (PSF) size storageunit 804 in addition to the above mentioned configuration. In the PSFsize storage unit 804, a size of a point spread function of the imagingoptical system 120 is stored, so as to correspond to the firstpositional shift amount.

The general flow of the depth calculation procedure of this modificationis the same as above, but, in the step of setting the size of the secondbase area in step S3-2 in FIG. 6B, the size of the second base area isset based on the PSF size outputted from the PSF size storage unit 804.In concrete terms, the base area setting unit 603 acquires the firstpositional shift amount from the positional shift amount calculationunit 602. Then, the base area setting unit 603 acquires the PSF sizecorresponding to the first positional shift amount from the PSF sizestorage unit 804. The base area setting unit 603 sets the size of thesecond base area in accordance with the PSF size acquired from the PSFsize storage unit 804.

In the depth calculation unit 102 shown in FIG. 8A, the area size of thesecond base area can be more appropriately set by setting the size ofthe second base area in accordance with the PSF size based on thedefocus of the imaging optical system 120. As a result, the area size ofthe second base area is not set too large (or too small), and anincrease of computation amount and positional shift amount error can beprevented.

The blur size of the imaging optical system 120 can be expressed by 3σ(three times the standard deviation σ) of PSF, for example. Therefore,the PSF size storage unit 804 outputs 3σ of PSF of the imaging opticalsystem 120 as the size of PSF. It is sufficient if the PSF size storageunit 804 stores the PSF size only for the central angle of view, but itis preferable to store the PSF size of the peripheral angle of view aswell if the aberration of the imaging optical system 120 at theperipheral angle of view is large. The PSF size may be expressed as afunction representing the relationship of the PSF size and thepositional shift amount, so that the coefficients are stored in the PSFstorage unit 804. For example, the PSF size may be calculated using alinear function in which a reciprocal number of the diaphragm value (Fvalue) of the imaging optical system is a coefficient, as shown inExpression 4, and the coefficients k1 and k2 may be stored in the PSFsize storage unit 804.

$\begin{matrix}{{PSFsize} = {{\frac{k\; 1}{F} \times {r}} + {k\; 2}}} & (4)\end{matrix}$Here PSFsize is a PSF size, r is a first positional shift amount, F isan F value of the imaging optical system 120, and k1 and k2 arepredetermined coefficients.

The PSF size may be determined as shown in Expression 5, consideringthat the ratio of the base line length (distance 513) described withreference to FIG. 5B and the diameter of the exit pupil 103 of theimaging optical system 120 are approximately the same as the ratio ofthe absolute value of the first positional shift amount and the PSFsize.

$\begin{matrix}{{PSFsize} = {{\frac{k\; 1 \times D}{w} \times {r}} + {k\; 2}}} & (5)\end{matrix}$Here, w is a base line length, D is a diameter of the exit pupil, and k1and k2 are predetermined coefficients. The size of PSF and the defocusamount have an approximate proportional relationship. If it isconsidered that the defocus amount and the positional shift amount havean approximate proportional relationship, as shown in Expression 3, thecoefficient k2 in Expression 4 and Expression 5 is not essentiallyrequired.

The base area setting unit 603 according to this modification sets thesize of the second base area in accordance with the PSF size acquiredfrom the PSF size storage unit 804. FIG. 8B is a graph of which abscissaindicates the PSF size, and the ordinate indicates the size of thesecond base area. As the solid line in FIG. 8B indicates, the size ofthe second base area in accordance with the blur of the acquired imagecan be set by setting the second base area and the PSF size to have aproportional relationship. Further, the PSF size and the size of thesecond base area may be set to have a proportional relationship if thePSF size exceeds the threshold 810, and may be set such that the size ofthe second base area becomes constant if the PSF size is less than thethreshold 810 as the broken line in FIG. 8B indicates.

In either case, the second base area is more appropriately set bysetting the second base area in accordance with the change of the PSFsize due to the defocus of the imaging optical system 120. Thereby, thearea size of the second base area is not set too large (or too small),and an increase of computation amount and positional shift amount errorcan be prevented.

<Modification 2 of Depth Calculation Unit>

As another modification of this embodiment, the depth calculation unit102 may include an imaging performance value storage unit instead of thePSF size storage unit 804. From the imaging performance value storageunit, a value representing the imaging performance of the object imageformed by the imaging optical system 120 is outputted. The imagingperformance can be expressed, for example, by an absolute value of theoptical transfer function (that is, a modulation transfer function, andhereafter called “MTF”), which indicates the imaging performance of theimaging optical system 120. In FIG. 8C, the abscissa indicates the firstpositional shift amount, and the ordinate indicates the MTF of theimaging optical system 120 at a predetermined spatial frequency. As theabsolute value of the first positional shift amount is smaller, thedefocus amount is smaller, and a higher MTF can be acquired. The basearea setting unit 603 may acquire the MTF corresponding to the firstpositional shift amount from the imaging performance value storage unit,as shown in FIG. 8D, and set a smaller second base area as the MTF ishigher. As the MTF of the imaging optical system 120 is higher, contrastof the object image deteriorates less, and a clearer image can beacquired. Therefore, the base area, considering the opticalcharacteristic of the imaging optical system 120, can be set by settingthe second base area based on the MTF corresponding to the firstpositional shift amount. Thereby, the second base area is not set toolarge (or too small), and an increase of the computation amount andpositional shift amount error can be prevented. The information storedin the PSF size storage unit 804 need not be information that indicatesthe size of the PSF, information that indicates the MTF, or informationthat indicates the optical characteristic of the imaging optical system120, as shown in modification 1 and modification 2 of this embodiment.Required is information indicating that the optical characteristic ofthe imaging optical system 120 is stored. By setting the second basearea in accordance with the blur size of the imaging optical system 120,using the information that indicates the optical characteristic of theimaging optical system 120 and the first positional shift amount, anincrease of the computation amount and positional shift amount error canbe prevented, and the depth to the object can be calculated at highaccuracy.

<Modification 3 of Depth Calculation Unit>

As another modification of this embodiment, the procedure shown in FIG.8E may be used. In the following description, it is assumed that thedepth calculation unit 102 includes a base area setting determinationunit in addition to the configuration shown in FIG. 6A.

In FIG. 8E, a base area setting determination step in step S3-4 and astep of setting the first positional shift amount as the secondpositional shift amount in step S3-5 are added to the procedure in FIG.6B. In step S3-4, determination processing is executed, in whichprocessing advances to step S3-2 if the absolute value of the firstpositional shift amount is greater than a predetermined threshold, orotherwise, advances to step S3-5. In step S3-5, processing to set thefirst positional shift amount as the second positional shift amount isexecuted.

In the procedure shown in FIG. 8E, only when the contrast of the imagedrops because the defocus amount is large, the second base area is setbased on the first positional shift amount. Then, the second positionalshift amount is calculated. If the defocus amount is small, the firstpositional shift amount is set as the second positional shift amount. Byfollowing this procedure, a number of pixels for which the positionalshift amount is calculated twice is decreased. Thereby, the computationamount can be further decreased while reducing the positional shiftamount error.

In the above description, it is assumed that the depth calculation unit102 includes the base area setting determination unit in addition to theconfiguration shown in FIG. 6A, but may include the base area settingdetermination unit in addition to the configuration shown in FIG. 8A. Inthis case, as well, a number of pixels for which the positional shiftamount is calculated twice is decreased, and the computation amount canbe further decreased.

<Other Examples of First Image Data and Second Image Data AcquisitionMethod>

In the first embodiment, two image data having different points of vieware acquired by splitting the luminous flux of one imaging opticalsystem, but two image data may be acquired using two imaging opticalsystems. For example, the stereo camera 1000 shown in FIG. 10 may beused. In the stereo camera 1000, two imaging optical systems 1020 and1021, two imaging elements 1010 and 1011, a depth calculation unit 102,an image storage unit 104, an image generation unit (not illustrated),and a lens driving control unit (not illustrated) are disposed inside acamera casing 130. The imaging optical systems 1020 and 1021, theimaging elements 1010 and 1011, the depth calculation unit 102 and theimage storage unit 104 constitute a depth calculation apparatus 110.

In the case of the stereo camera, it is assumed that the image datagenerated by the imaging element 1010 is the first image data, and theimage data generated by the imaging element 1011 is the second imagedata. The optical characteristic of the imaging optical system 1020 andthat of the imaging optical system 1021 are preferably similar. Thedepth calculation unit 102 can calculate the depth to the objectaccording to the depth calculation procedure described with reference toFIG. 3. The base line length in the case of the stereo camera 1000 canbe the depth between the center position of the exit pupil of theimaging optical system 1020 and the center position of the exit pupil ofthe imaging optical system 1021. To convert the second positional shiftamount, which is calculated according to the positional shift amountcalculation procedure described with reference to FIG. 6B, into thedepth to the object, a known method can be used.

In this modification, as well, the changes of the positional shiftamount in the base area and the influence of noise generated uponacquiring images can be reduced by setting the size of the second basearea considering the optical characteristic of the imaging opticalsystems 1020 and 1021. Particularly, in the case of the stereo camera1000, the F values of the imaging optical systems 1020 and 1021 must besmall in order to acquire high resolution images. In this case, a dropin contrast due to defocus becomes conspicuous. Hence, the depthcalculation apparatus that includes the depth calculation unit 102according to this embodiment can ideally calculate the depth to theobject.

The above mentioned depth calculation apparatus according to the firstembodiment can be installed by software (programs) or hardware. Forexample, a computer program is stored in memory of a computer (e.g. amicrocomputer, a CPU, an MPU, an FPGA) included in the imaging apparatusor image processing apparatuses, and the computer executes the programto implement each processing. It is also preferable to dispose adedicated processor, such as an ASIC, to implement all or a portion ofthe processing of the present invention using logic circuits. Thepresent invention is also applicable to a server in a cloud environment.

The present invention may be implemented by a method constituted bysteps to be executed by a computer of a system or an apparatus, whichimplements the above mentioned functions of the embodiment by readingand executing a program recorded in a storage apparatus. For thispurpose, this program is provided to the computer via a network, or viavarious types of recording media that can function as a storageapparatus (that is, a computer readable recording media that holds datanon-temporarily), for example. Therefore, this computer (including sucha device as a CPU and an MPU), this method, this program (includingprogram codes and program products), and the computer readable recordingmedia that non-temporarily stores this program are all included withinthe scope of the present invention.

Embodiment(s) of the present invention can also be realized by acomputer of a system or an apparatus that reads out and executescomputer executable instructions (e.g., one or more programs) recordedon a storage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., an application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., a central processingunit (CPU), or a micro processing unit (MPU)) and may include a networkof separate computers or separate processors to read out and to executethe computer executable instructions. The computer executableinstructions may be provided to the computer, for example, from anetwork or the storage medium. The storage medium may include, forexample, one or more of a hard disk, a random-access memory (RAM), aread only memory (ROM), a storage of distributed computing systems, anoptical disk (such as a compact disc (CD), a digital versatile disc(DVD), or a Blu-ray Disc (BD)™), a flash memory device, a memory card,and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

The invention claimed is:
 1. A positional shift amount calculationapparatus that calculates a positional shift amount, which is a relativepositional shift amount between a first image based on a luminous fluxthat has passed through a first imaging optical system, and a secondimage, the apparatus comprising: at least one processor operativelycoupled to a memory to functions as: (a) a calculation unit adapted tocalculate a positional shift amount based on data within a predeterminedarea out of first image data representing a first image and second imagedata representing a second image; and (b) a setting unit adapted to seta relative size of the area to the first and second image data, wherein(i) the calculation unit is adapted to calculate a first positionalshift amount using the first image data and the second image data in thearea having a first size that is preset, (ii) the setting unit isadapted to set a second size of the area based on the size of the firstpositional shift amount and an optical characteristic of the firstimaging optical system, and (iii) the calculation unit is adapted tocalculate a second positional shift amount using the first image dataand the second image data in the area having the second size, andwherein, when an absolute value of the first positional shift amount isgreater than a predetermined threshold, the setting unit sets the secondsize to be larger as the absolute value of the first positional shiftamount is greater.
 2. The positional shift amount calculation apparatusaccording to claim 1, wherein the setting unit sets the second size bychanging sizes of the first and second images.
 3. An imaging apparatuscomprising: an imaging optical system; an imaging element adapted toacquire image data based on a luminous flux that has passed through theimaging optical system; and the positional shift amount calculationapparatus according to claim
 1. 4. The positional shift amountcalculation apparatus according to claim 1, wherein, when the absolutevalue of the first positional shift amount is greater than thepredetermined threshold, the setting unit sets the second size and thecalculation unit calculates the second positional shift amount, and,when the absolute value of the first positional shift amount is thepredetermined threshold or less, the first positional shift amount isset as the second positional shift amount.
 5. The positional shiftamount calculation apparatus according to claim 1, wherein the opticalcharacteristic of the first imaging optical system includes one of adiaphragm value of the first imaging optical system, a size of a pointspread function of the first imaging optical system, and an imagingperformance of the first imaging optical system.
 6. The positional shiftamount calculation apparatus according to claim 1, wherein the firstimage is an image based on a luminous flux that has passed through afirst pupil area in an exit pupil of the first imaging optical system,and the second image is an image based on a luminous flux that haspassed through a second pupil area, which is different from the firstpupil area, in the exit pupil of the first imaging optical system. 7.The positional shift amount calculation apparatus according to claim 1,wherein the first image is an image based on the luminous flux that haspassed through the first imaging optical system, and the second image isan image based on a luminous flux that has passed through a secondimaging optical system that is different from the first imaging opticalsystem.
 8. The positional shift amount calculation apparatus accordingto claim 6, further comprising a storage unit in which a point spreadfunction (PSF) size of the first imaging optical system, is stored,wherein the setting unit acquires the PSF size corresponding to thefirst positional shift amount from the storage unit, and sets the secondsize based on the acquired PSF size.
 9. The positional shift amountcalculation apparatus according to claim 8, wherein the PSF size isgiven by the following expression, where r is the first positional shiftamount, F is a diaphragm value of the first imaging optical system, andk1 and k2 are predetermined coefficients:${PSFsize} = {{\frac{k\; 1}{F} \times {r}} + {k\; 2.}}$
 10. Thepositional shift amount calculation apparatus according to claim 8,wherein the PSF size is three times a standard deviation of the pointspread function corresponding to the first positional shift amount. 11.The positional shift amount calculation apparatus according to claim 6,further comprising a storage unit in which a point spread function PSFsize of the first imaging optical system, is stored, wherein the settingunit acquires the PSF size corresponding to the first positional shiftamount from the storage unit, and sets the second size based on theacquired PSF size, and the PSF size is given by the followingexpression, where r is the first positional shift amount, w is a depthbetween a center of gravity position of the first pupil area and acenter of gravity position of the second pupil area, D is a diameter ofthe exit pupil of the imaging optical system, and k1 and k2 arepredetermined coefficients:${SFsize} = {{\frac{k\; 1 \times D}{w} \times {r}} + {k\; 2.}}$ 12.The positional shift amount calculation apparatus according to claim 6,further comprising a storage unit, in which an evaluation valuerepresenting imaging performance of the first imaging optical system isstored, wherein the setting unit acquires the evaluation valuecorresponding to the first positional shift amount from the storageunit, and sets the second size based on the acquired evaluation value.13. The positional shift amount calculation apparatus according to claim12, wherein the evaluation value indicates an optical transfer functionof the first imaging optical system at a predetermined spatialfrequency.
 14. The positional shift amount calculation apparatusaccording claim 1, further comprising a depth conversion unit adapted toconvert the second positional shift amount, calculated by thecalculation unit, into a defocus amount that is a depth from anestimated focal position to a focal position of an imaging opticalsystem, based on a predetermined conversion coefficient.
 15. Anon-transitory computer-readable storage medium storing a computerprogram, when run by a computer, causes the computer to execute: a firstcalculation step of calculating a first positional shift amount based ondata within an area having a predetermined first size, out of firstimage data representing a first image and second image data representinga second image, wherein the first image is based on a luminous flux thathas passed through a first imaging optical system; a setting step ofsetting a second size, which is a relative size of the area to the firstand second image data, based on the size of the first positional shiftamount and an optical characteristic of the first imaging opticalsystem; and a second calculation step of calculating a second positionalshift amount using the first image data and the second image data in thearea having the second size, wherein, when an absolute value of thefirst positional shift amount is greater than a predetermined threshold,the setting unit sets the second size to be larger as the absolute valeof the first positional shift amount is greater.
 16. A positional shiftamount calculation method for a positional shift amount calculationapparatus to calculate a positional shift amount, which is a relativepositional shift amount between a first image based on a luminous fluxthat has passed through a first imaging optical system, and a secondimage, the method comprising: a first calculation step of calculating afirst positional shift amount based on data within an area having apredetermined first size, out of first image data representing the firstimage and second image data representing the second image; a settingstep of setting a second size, which is a relative size of the area tothe first and second image data, based on the size of the firstpositional shift amount and an optical characteristic of the firstimaging optical system; and a second calculation step of calculating asecond positional shift amount using the first image data and the secondimage data in the area having the second size, wherein, when an absolutevalue of the first positional shift amount is greater than apredetermined threshold, the setting unit sets the second size to belarger as the absolute vale of the first positional shift amount isgreater.